We consider the Casimir force betweeen two dielectric bodies described by the plasma model and between two infinitely thin plasma sheets. In both cases in addition to the photon modes surface plasmons are present in the spectrum of the electromagnetic field. We investigate the contribution of both types of modes to the Casimir force and confirm resp. find in both models large compensations between the plasmon modes themselves and between them and the photon modes especially at large distances. Our conclusion is that the separation of the vacuum energy into plasmon and photon contributions must be handled with care except for the case of small separations.